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Summary 

With  the  changing  character  of  warfare,  information  superiority  is  a  high  priority.  Our  decision  makers 
must  be  able  to  accurately  assess  the  situation,  decide  upon  a  course  of  action,  and  set  that  course  of  action 
in  motion  before  the  adversary  can  react.  It’s  always  a  challenge  to  provide  decision  makers  with  the  right 
information  in  an  easily  understandable  format.  This  is  particularly  true  as  more  and  more  information  can 
be  made  available.  It  becomes  very  important  to  display  the  right  information  at  the  right  time  and  to  make 
it  easy  to  find  and  understand.  Today’s  new  computer  and  display  technologies  afford  us  a  broad  array  of 
options  for  information  presentation.  Our  challenge  is  to  develop  display  interfaces  that  deliver  decision- 
quality  information  culled  from  various  sensors/systems  directly  to  decision-makers. 

Decision  makers  want  to  understand  the  uncertainties  associated  with  the  information  provided  to  them. 
New  display  techniques  are  needed  to  enable  the  decision  maker  to  understand  the  meta-data  associated 
with  information.  Some  examples  of  meta-data  include: 

•  Where  did  the  information  come  from? 

•  What  types  of  uncertainty  are  associated  with  the  data? 

•  Sensor  precision  &  range  limits 

•  False  variation  due  to  noise 

•  Human  observers’  limits  of  perception  or  trust/reliability 

•  How  old  is  the  data? 

•  Has  the  data  been  filtered  by  models?  What  might  have  been  lost  in  the  filtering  process? 

Air  Force  Research  Laboratory  Human  Effectiveness  Directorate  researchers  are  working  to  develop  and 
validate  a  new  generation  of  icons  for  battlespace  visualization.  These  “knowledge  glyphs”  will  portray 
multiple  dimensions  of  information  as  well  as  the  uncertainty  associated  with  each.  There  are  a  number  of 
research  issues  to  be  addressed: 

•  How  many  dimensions  can  be  displayed  in  one  glyph  and  be  readily  understandable  by  the 
user? 

•  How  can  the  uncertainties  associated  with  information  be  displayed? 

•  Should  glyphs  be  2D,  3D,  or  4D? 

•  Should  all  information  associated  with  a  dimension  be  displayed  at  once,  or  should  there  be 
drill  down? 

•  Should  glyphs  be  temporally  dynamic? 

•  How  will  the  perception  of  information  change  as  a  function  of  screen  resolution? 

•  Can  glyphs  be  created  that  counter  information  bias? 

Decision-makers  must  be  able  to  visualize  the  battlespace  in  order  to  assess  current  and  future  situations. 
Information  portrayal  must  support  this  process  and  promote  more  rapid  assimilation  of  data.  This  requires 
a  systems  engineering  approach,  in  which  the  decision-maker  is  treated  as  part  of  the  system  along  with 
computer  and  display  technologies.  There  are  many  new  and  creative  concepts  for  display  of  information. 
However,  very  few  have  been  tested.  Laboratory  experimentation  is  needed  to  optimize  and  validate  new 
display  concepts  for  specific  applications. 
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I’m  from  the  Battlespace  Visualization  Branch  of  HECV.  Our  branch  conducts 
research  in  the  two  broad  areas  of  Visualization  and  Vision  Enhancement  for  a 
variety  of  Air  Force  applications,  including  cockpits  and  command  and  control 
centers. 


Visualization  of  Uncertainty 

Problem  Statement 


*  Current  military  displays  do  not  provide  decision¬ 
makers  with  all  the  information  they  need 

*  Displayed  information  is  not  always  intuitive 

*  Decision-makers  would  also  like  to  understand  the 
uncertainties  associated  with  the  information  that  is 
provided  to  them 

—  Age  of  information 
—  Reliability  of  source 
—  Conflicting  data 
—  Etc. 
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It’s  always  a  challenge  to  provide  decision  makers  with  the  right  information 
in  an  easily  understandable  format.  This  is  particularly  true  as  more  and  more 
information  can  be  made  available.  It  becomes  very  important  to  display  the 
right  information  at  the  right  time  and  to  make  it  easy  to  find  and  understand. 


Particularly  in  the  area  of  command  and  control,  decision  makers  want  to 
know  the  uncertainties  associated  with  the  information  they  are  using.  A  few 
types  of  uncertainty  are  listed  here. 


Here  is  a  map  with  some  examples  of  current  military  symbology.  These 
symbols  can  show  as  many  as  10  different  dimensions  of  information  about  an 
entity.  However,  they  do  very  little  to  address  uncertainty.  For  example  the 
circled  icons  represent  air  tracks.  The  yellow  color  and  irregular  shape 
indicate  that  they  are  unknowns.  The  icons  in  the  square  boxes  on  the  lower 
left  represent  aircraft  (the  green  symbols  are  neutral  rotary  wing  and  the  red 
are  hostile  fixed  wing.  The  dotted  line  surrounding  the  symbol  indicated  that 
it  is  expected  or  planned  as  opposed  to  actual.  These  are  the  only  instances  in 
which  this  symbology  set  addresses  uncertainty. 


Visualization  of  Uncertainty 

Challenges  and  Opportunities 


Challenge: 

“Stop  concentrating  on  individual  systems  and  start 
focusing  on  the  information  they  provide  . . .  Automate 
processing  so  that  information  can  be  displayed 
intuitively.  In  doing  that,  operators  can  make  better 
decisions  more  quickly  . . .  The  sum  of  the  wisdom  is  a 
cursor  over  the  target . . .  Want  to  see  interfaces  that 
deliver  decision-quality  information  culled  from 
various  sensors/systems  directly  to  decision-makers” 

•  Source:  General  John  Jumper,  CSAF  at  C2ISR 
Summit,  AF  Magazine,  June  02 
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As  stated  here  by  Gen  Jumper,  our  challenge  is  to  deliver  decision  quality 
information  to  decision  makers. 


All  of  you  are  probably  familiar  with  the  OODA-loop.  The  goal  is  to  cycle 
through  the  process  of  Observe-Orient-Deciding- Acting  faster  than  the 
adversary.  Better  visualization  will  help  decision-makers  to  make  the  right 
decision  faster. 


i  k 


Visualization  of  Uncertainty 

Challenges  and  Opportunities 


Opportunities: 

*  More  Computer  Processing  Power 

*  Increasing  Bandwidth 

*  Greater  Graphics  Capabilities 

*  New  Display  Technologies 


Not  so  many  years  ago,  our  options  for  displaying  information  were  text  or 
tabular  displays  or  analog  instruments.  Today’s  new  and  improved 
technologies  afford  us  a  broad  array  of  options  for  information  presentation. 
But,  more  is  not  always  better.  We  must  be  sure  to  design  our  displays  with 
human  perceptual  and  cognitive  capabilities  and  limitations  in  mind. 


Visualization  of  Uncertainty 

Goal 


Develop  visualization  concepts  to  provide  decision 
makers  the  information  they  need  in  a  format  that  is 
quickly  and  easily  understandable 

•  Approach: 

—  Explore  information  portrayal  capabilities  of 
multidimensional  icons 

—  Explore  information  portrayal  capabilities  of 
dynamic  icons 

—  Explore  other  methods  for  portraying  uncertainty 


Other  methods  might  include  drill  down  to  secondary  icons  or  textual 
information. 
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Visualization  of  Uncertainty 

Approach 


Focus  -  Common  Operating  Picture 
Multiple  map  layers 
ID  important  entities  for  each  layer 
ID  information  requirements  for  each 
ID  associated  uncertainties 

Evaluate  adaptation  potential  of  current  symbology  as  well  as 
other  visualization  techniques 

Evaluate  applicability  of  new  display  technologies 

Develop  visualization  concepts 

Test  experimentally 


We’re  focusing  on  a  common  operating  picture  (COP),  because  everyone 
seems  to  agree  that  one  is  needed.  However,  there  is  not  a  lot  of  agreement  as 
to  what  exactly  should  be  included  in  the  COP.  First,  we  need  to  look  at  the 
user.  What  is  his  function.  What  tasks  does  he  need  to  perform.  What 
information  does  he  need  to  perform  each  task.  We  start  with  a  map,  because 
a  geospatial  foundation  is  needed  on  which  to  overlay  other  information.  A 
layered  map  is  just  one  concept  -  others  include  various  declutter  mechanisms 
or  a  lens  for  getting  greater  detail  in  a  given  area. 


After  we’ve  identified  all  the  dimensions  of  information  required,  we  can  then 
identify  the  uncertainties  associated  with  each  information  dimension. 


New  display  technologies,  such  as  depth  displays  and  3D  displays  may  offer 
potential  for  information  display. 


Experimental  testing  of  new  visualization  concepts.  There  are  many  new  ideas 
for  display  of  information.  However,  very  few  have  been  tested  and  validated 
for  specific  applications. 


Here’s  just  one  display  concept  for  the  common  operating  picture.  All  layers 
of  the  map  are  georeferenced.  Any  layer,  with  it’s  appropriate  detailed 
information  can  be  accessed. 


Here  are  just  two  display  concepts  for  future  command  centers. 


&  Visualization  of  Uncertainty 

Display  Technology  Evaluations 


•Sharp  3D  display 

Software  and  special 
screen  =  3D 

But.. .not  without  problems 
Can  be  hard  on  the 
eyes 

The  “eye  box”  is  quite 
small 

No  one  has  really 
shown  the  benefit  of 
3D  especially  as 
compared  to  2D  for 
complex  scenes 


Immersive/VR  is  still  in  a  very 
embryonic  stage 

—  More  concerned  with  “fun” 
than  functionality 

—  How  do  you  interact  with 
information  that  is  all 
around? 

*  3D  audio? 

*  Visual  cues? 

•  Tactile  cues? 

•  Ail  of  the  above? 


There  are  a  number  of  new  display  technologies  coming  on  the  market .  These 
look  exciting  and  may  have  potential  for  military  applications.  However,  there 
are  a  number  of  issues  to  be  resolved.  Laboratory  testing  is  needed  to  assess 
the  potential  usefulness  of  these  displays  for  particular  applications. 
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Another  exciting  display  technology,  but  how  do  we  interact  with  it? 


Up 


Visualization  of  Uncertainty 

Beyond  Mil-Std-2525  Symbology 


•  New  display  techniques  are  needed  to  enable  the 
decision  maker  to  understand  the  meta-data 
associated  with  information 

—  Where  did  the  information  come  from? 

—  What  types  of  uncertainty  are  associated  with 
the  data? 

*  Sensor  precision  &  range  limits 

*  False  variation  due  to  noise 

*  Human  observers’  limits  of  perception  or 
trust/reliability 

—  How  old  is  the  data? 

—  Has  the  data  been  filtered  by  models?  What 
might  have  been  lost  in  the  filtering  process? 
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We  need  a  single  symbol/icon  which  portrays  multiple  dimensions  of 
information  as  well  as  the  uncertainty  associated  with  each 


Applicable  Joint  Forces  Command  S&T  Focus  Areas  include: 

Effects  Based  Operations  Visualization  Tools 
3D  Sensor  Fusion 

Persistent  and  Pervasive  Intelligence  Surveillance  and  Reconnaissance 
Predictive  Battlespace  Analysis 
Joint  Urban  Warfare 


Visualization  of  Uncertainty 

Knowledge  Glyphs 


Develop  next  generation  of  icons 

*  Temporally  dynamic  graphic 
objects  that  integrate  and 
encode  multiple  types  of 
information 

*  Temporal  variability  and  graphic 
representation  enable  the  glyph 
to  express  various  types  of 
uncertainty 

*  Readily  understandable  by  the 
user 


ATC-NY 
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Some  concepts  for  Knowledge  Glyphs 


Visualization  of  Uncertainty 

Knowledge  Glyph  Research  Issues 


2D,  3D,  or  4D? 

How  can  the  uncertainties  associated  with  information  be 
displayed? 

How  many  dimensions  can  be  displayed  in  one  glyph? 

How  many  dimensions  can  a  human  perceive  at  one  time? 

Should  all  information  associated  with  a  dimension  be  displayed 
at  once,  or  should  there  be  drill  down? 

How  will  the  perception  of  information  change  as  a  function  of 
screen  resolution? 

Can  glyphs  be  created  that  counter  information  bias? 


There  are  a  number  of  research  issues  that  must  be  addressed  in  the  creation  of 
Knowledge  Glyphs. 


4  M  Visualization  of  Uncertainty 

Improved  Decision-Making  Using  Complex  Icons ^ 


AFOSR-funded  basic  research 


Objectives:  To  Answer  the  Following  Basic  Research  Questions 

(1)  How  many  elements  of  information  can  effectively  be  conveyed 
by  a  single  icon? 

(2)  How  can  associated  information  uncertainty  be  effectively 
conveyed? 

(3)  Utility  of  dynamic  icons? 

Issues: 

(a)  Icon  selection  or  design 

(b) What  kind  of  information  should  be  portrayed  with  an  icon 


HEC  has  received  funding  from  the  Air  Force  Office  of  Scientific  Research  to  look  at 
some  basic  research  issues  related  to  complex  icons. 

Elements  of  information  that  might  be  portrayed  on  an  icon  include: 

Location 

Size 

Identity 

Friend  vs  foe 

Association 

capability 


Visualization  of  Uncertainty 
Improved  Decision-Making  Using  Complex  Icons 

Select  Icons  for  Evaluation 

(1)  Mil-Std  2525  -  Dates  back  to  Napoleon. 


(2)  In  1973- Chernoff  Faces. 


(3)  Castles/  Houses. 


(4)  Polytopes  (star  glyphs,  radar  charts). 


(5)  Intuitive  Objects  (apple). 


Some  examples  of  multidimensional  icons. 


Here  are  just  some  of  the  characteristics  of  icons  that  might  be  applied  to 
coding  various  dimensions  of  information.  Some  may  be  more  salient  than 
others. 
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Improved  Decision-Making  Using  Complex  Icons 


Mil-Std-2525  Symbology:  2  -  2.5  D  of  Information 


BATTLE 
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Color  is  used  to  code  affiliation. 

Shape  is  used  to  depict  battle  dimension  -  surface  vs  space  or  air  vs  sub 
surface. 

Installation  is  depicted  by  adding  another  shape. 


We  might  depict  affiliation  and  battle  dimension  intuitively  using  chemoff 
faces.  Adding  more  dimensions  of  information  would  be  more  difficult. 
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Improved  Decision-Making  Using  Complex  Icons' 


Intuitive  Apple  Coding  Dimensions 


•  Color  -  age  of  data 

•  Shiny-sensors  vs  rough -database 

•  Stem  length  -  fuel 

•  Stem  width  -  armed 

•  Movement 

•  Cloud  -  conflicting  data 


FipM  5  -  All  Irfcjitiv*  Icon  to  Otptay  10  or  Mow  Complex  Oimwwior*  h  tfw  Data 
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Here’s  another  attempt  at  encoding  information  dimensions  intuitively.  This 
representation  was  developed  by  Dr.  Dan  Repperger  at  AFRL. 
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Improved  Decision-Making  Using  Complex  lcons|l^ 


Information  Theory  Analysis 


We  model  the  human  as  an  information  channel.  We  calculate  the  effective 
information  throughput  as  a  function  of  icon  complexity. 


H«« 

Input 

Information 


H(x/y)-Lo*t 
Information  ■ 


.  Eqtfvccation 

H0)« 

J  T^y)  *  TrwBm«1«<J 
information 

Output 

Information 

% 

Hfy&) »  Not**  or 

Spurious  Information 


Figure  1  -  Basic  Elements  of  the  Information  Channel 


H(y)  (output  Information)  or 


Figure  2  -  Hypothesized  Shape  of  T(x;y)  or  H(y)  versus  n  *  number  of  Icon  dimensions 

(cf.  Hick,  Hyman  Law  the  precursor  to  Fitts’  Law).  * 


The  studies  in  information  theory  involving  humans  can  be  traced  to  one  of  the 
first  published  works  in  this  area  by  Merkel  (1885).  Using  strip  chart  recorders 
and  other  crude  time  measurement  devices,  in  his  early  experiments,  he  noted 
that  the  time  to  complete  a  task  with  a  number  of  alternatives  or  choices  was 
always  longer  if  the  number  of  choices  increased.  In  particular,  he 
distinguished  between  the  time  to  perform  a  pure  reaction  task  (no  choices) 
and  the  time  to  make  a  decision  on  a  complex  task  and  then  to  react.  The 
distinction  was  made  between  reaction  time  and  decision  time.  The  decision 
time  was  defined  as  the  difference  between  the  total  time  to  complete  the 
complex  task  and  the  pure  reaction  time. 

C.  E.  Shannon’s  seminal  book  (Shannon  (1949))  is  sometimes  viewed  as  the 
birth  of  information  theory  when  he  documented  8  axioms  that  define  what 
makes  up  an  information  channel.  Shannon  emphasized  using  the  unit  of  a  bit 
or  log  base  2  of  the  number  of  choices,  which  eventually  became  the  standard 
to  measure  information.  Work  then  began  in  studying  how  humans  process 
information.  At  Cambridge,  England,  W.  E.  Hick  publicized  (Hick  (1952))  a 
linear  relationship  between  human  reaction  time  to  the  logarithm  of  the 
number  of  alternatives  (equally  likely  alternatives).  In  the  next  year,  R. 

Hyman,  from  the  Johns  Hopkins  University  in  the  USA  documented  a  similar 
result  (Hyman  (1953)).  This  became  known  as  the  Hick-Hyman  Law  and  deal 
with  the  “perceptual  loop”  aspects  of  human  information  processing.  Again, 
the  time  for  a  human  to  reaction  to  a  complex  perceptual  stimulus  was 
proportional  to  the  logarithm  of  the  number  of  alternatives  of  the  complex 
stimuli.. 


The  information  overload  point  is  calculated  as  shown  here.  As  the  graph  in 
the  lower  right  shows,  we  can  determine  the  number  of  information 
dimensions  that  will  result  in  information  overload.  Information  overload  is 
said  to  occur  when  information  begins  to  be  lost  in  the  transfer  process. 
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Visualization  of  Uncertainty 
Improved  Decision-Making  Using  Complex  Icon 


Accomplishments  to  Date: 

•  Pilot  study  using  Mil-Std  2525  symbology  -  10  dimensions 

•  Preliminary  findings  -  information  overload  occurs  beyond  4-5 
dimensions  of  information 


The  way  ahead: 

•  Explore  saliency  of  various  coding  dimensions 

•  Explore  other  symbol  sets 

•  Explore  effects  of  dynamics 
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We’ve  just  completed  a  pilot  study  using  Mil-Std  2525  symbology  with  up  to 
10  dimensions  of  information.  We  found  that  for  this  particular  symbol  set 
and  coding,  information  transfer  tended  to  deteriorate  if  more  than  4  or  5 
dimensions  of  information  were  displayed  at  once. 


We’ve  seen  that  some  coding  techniques,  such  as  color  and  shape,  facilitate 
information  transfer  better  than  others.  This  will  be  explored  further.  We’ll 
also  look  at  other  symbol  sets  to  see  if  the  same  rules  apply. 


Then  we’ll  explore  the  effects  of  dynamic  coding  dimensions. 


Visualization  of  Uncertainty 

Collaborations 

•  Participation  with  NGA  and  NSA  Technology  Futures  Collaboratory 

—  Visualization  of  Intelligence  Information 

•  Participation  in  Visualization  of  Uncertainty  Workshop 

—  National  Academy  of  Sciences 
—  National  Security  Agency 
—  Academia 

•  Collaboration  with  AFRUIF  -  C2  Concepts  for  a  New  Generation 

•  Collaboration  within  AFRL  Human  Effectiveness  Directorate 

—  Cognitive  Systems  (HECS)  /  Battlespace  Visualization  (HECV)/ 
Collaborative  Systems  (HECP) 

•  Knowledge  Glyphs  for  CPE 

•  Complex  Icons  and  Information  Theory 


Visualization  of  uncertainty  is  receiving  a  good  deal  of  attention.  These  are 
just  some  of  our  collaborations. 


Visualization  of  Uncertainty 

Summary 


Decision-makers  must  mentally  Visualize  to  assess  current  and 
future  situations 

Information  portrayal  must  support  this  process  and  promote 
more  rapid  assimilation  of  data 

Requires  a  systems  engineering  approach 

—  Computer  and  display  technologies 

—  Decision-maker  as  part  of  system 

*  Perceptual  capabilities  /  limitations 

•  Cognition  /  Decision  Theory 

—  Mission  Requirements  -  Measures  of  Effectiveness 


In  dealing  with  complex  information  transfer,  we  need  visualization 
techniques  that  help  the  decision  maker  to  quickly  assess  and  understand  the 
situation. 


We  emphasize  a  systems  engineering  approach  where  the  decision  maker  is 
part  of  the  system,  as  well  as  the  computer  and  display  technologies.  We  need 
to  design  the  system  with  the  human’s  perceptual  capabilities  and  limitations 
in  mind  as  well  as  understand  how  humans  assimilate  and  use  information  to 
make  decisions.  And,  of  course,  we  must  always  stay  focused  on  the  mission 
requirements  and  how  we  can  quantify  the  effectiveness  of  any  visualization 
technique. 


